feat(crashdump): Add crash dump functionality for fatal errors #1594
Conversation
|
Another approach I realized after publishing could be to move the GameMemory allocations from using GlobalAlloc to instead create a separate GameMemory heap and use HeapAlloc. |
| { | ||
| // Find the full path to the dbghelp.dll file in the system32 dir | ||
| GetSystemDirectory(m_sysDbgHelpPath, MAX_PATH); | ||
| strlcat(m_sysDbgHelpPath, "\\dbghelp.dll", MAX_PATH); |
There was a problem hiding this comment.
Agreed, I'll hold off on this one (but push changes for the review comments) until after #1066 is merged, then replace the loading functionality here with DbgHelpLoader and extend it with the MiniDumpWriteDump function.
| m_stacktrace[0] = NULL; | ||
| } | ||
| #endif | ||
| } |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
Pushed a commit attempting to address most of the comments in the previous review:
In addition:
Remaining: |
|
|
||
| // Use placement new on the process heap so TheMiniDumper is placed outside the MemoryPoolFactory managed area. | ||
| // If the crash is due to corrupted MemoryPoolFactory structures, try to mitigate the chances of MiniDumper memory also being corrupted | ||
| TheMiniDumper = new (::HeapAlloc(::GetProcessHeap(), HEAP_GENERATE_EXCEPTIONS, sizeof(MiniDumper))) MiniDumper; |
There was a problem hiding this comment.
After #1066 is merged, you can also use sysAllocateDoNotZero / sysFree for it.
| } | ||
|
|
||
| int dumpTypeFlags = MiniDumpWithIndirectlyReferencedMemory | MiniDumpScanMemory; | ||
| if (dumpType != DUMP_TYPE_MINIMAL) |
There was a problem hiding this comment.
Could also use switch case with fall through.
int dumpTypeFlags = 0;
switch (dumpType)
{
case DUMP_TYPE_FULL:
dumpTypeFlags |= MiniDumpWithFullMemory;
FALLTHROUGH;
case DUMP_TYPE_GAMEMEMORY:
dumpTypeFlags |= MiniDumpWithDataSegs | MiniDumpWithHandleData | MiniDumpWithThreadInfo | MiniDumpWithFullMemoryInfo;
FALLTHROUGH;
case DUMP_TYPE_MINIMAL:
dumpTypeFlags |= MiniDumpWithIndirectlyReferencedMemory | MiniDumpScanMemory;
break;
}| // Only include data segments for the game and ntdll modules to keep dump size low | ||
| if (output.ModuleWriteFlags & ModuleWriteDataSeg) | ||
| { | ||
| if (::StrCmpIW(input.Module.FullPath, m_executablePath) != 0 && !::StrStrIW(input.Module.FullPath, L"ntdll.dll")) |
3 participants
This change introduces the option to generate crash dumps, aka. mini dumps, on fatal errors.
The main minidump functionality is done by explicitly loading the
dbghelp.dllfrom the system directory, as the dbghelp.dll that is bundled with the game is an older version that does not include this functionality. There is an option to create small dumps or extended info dumps, currently both are created.Small dumps
These mostly contain stacks for the process threads and some stack variables, or to create dumps with extended info. The use case for these is to quickly determine where a crash occured, the type of crash, if it was already fixed etc. In addition, if the memory allocation structures are corrupted enough, an extended info dump might not succeed while the small dump should. The size of these dumps are typically on the order of 250kB.
Extended info
These contain global values, along with the memory regions allocated via the memory pool factory and the dynamic memory allocator. This makes all in-game objects available to the person debugging the crash dump, so for example
dt generalszh!TheWritableGlobalDatain WinDbg will show the state at the time the dump was created.An alternative option could be to not traverse the memory structures "manually" to get to the allocations and instead just specify the
MiniDumpWithFullMemoryflag toMiniDumpWriteDump, but that increases the file size considerably.As an example, dump of the generalszh process in the main menu with the shell map in the background yields a ~140MB dump when traversing and ~420MB with
MiniDumpWithFullMemory. Beyond that, the ~140MB file compresses to ~20MB with 7Z, so should be relatively easily transferable.Storage Location
Crash dumps are stored in a new folder called 'CrashDumps' under the userDir ("Documents\Command and Conquer Generals Zero Hour Data"), and on startup it will create this directory if it doesn't exist and delete any older dumps so only the 10 newest small and 2 newest extended info dumps are left. This is to preserve disk space, as the extended info files can be several hundred MB.
Integration points
For VS2022 builds, unhandled exceptions end up in the
UnhandledExceptionFilterin WinMain, which then get a reference to the actual exception that occurred and includes that in the dump.For VC6 builds, unhandled exceptions are caught in the
catch(...)blocks ofGameEngine::executewhich then calls RELEASE_CRASH. As there is no exception data available in this case to populate _EXCEPTION_POINTERS from, an intentional exception is triggered to get the trace of the current thread. This makes the stack traces for VC6 a bit more cryptic than VS2022 builds as the C++ exception handling gets included in the trace.Limitations
In the longer run we'll probably want to replace this code with a more mature solution, like CrashPad, but that currently depends on a newer compiler than VC6.
As the code is intended to be temporary, it's kept behind a new CMake feature so it can be easily removed. There are also some other decisions made with this in mind: